Comments on 'Adaptive increase in sample size when interim results are promising: a practical guide with examples' by C. R. Mehta and S. J. Pocock.

In their paper [1], Drs. Mehta and Pocock illustrate the use of a particular approach to revising the maximal sample size of a randomized clinical trial (RCT) by using an interim estimate of the treatment effect. Slightly extending the results of Gao, Ware, and Mehta [2], the authors define conditions on an adaptive rule such that one can know that the naive statistical hypothesis test that ignores the adaptation is conservative. They then use this knowledge to define an adaptive rule for a clinical trial. In our review of this paper, however, we do not find that such an adaptive rule confers any advantage by the usual criteria for clinical trial design. Rather, we find that the designs proposed in this paper are markedly inferior to alternative designs that the authors do not (but should) consider. By way of full disclosure, the first author of this commentary provided to the authors a signed referee’s report on an earlier version of this manuscript, and that report contained the substance (and most of the detail) of this review. In the comments to the editor accompanying that review, the first author described the dilemma that arose during that review. In essence, the methods described in the manuscript do not seem to us worthy of emulation. But on the other hand, the purpose of case studies in the statistical literature is to present an academic exposition of lessons that can be learned. From years of recreational spelunking, we have noted parallels between research and cave exploration. In both processes, explorers spend their time in the dark exploring the maze of potential leads, most often without a clear idea of where they will end up. Because the overwhelming majority of such leads are dead ends, the most useful companions to have along with you are the ones who will willingly explore the dead ends. However, they rapidly become the least useful companions if they have a tendency to explore the dead ends and then come back and tell you the leads went somewhere. Furthermore, the most important skill that any explorers can have is the ability to recognize when they are back at the beginning, lest they believe that the promising lead took them someplace new and become hopelessly lost. According to these criteria, then, the fact that we would not adopt some approach does not necessarily detract from the importance of a paper to the statistical literature. Instead, a paper’s value relates to the extent to which it contributes to our understanding of the methods, which can often be greatly enhanced by identifying dead ends and/or leads that take us back to the beginning. We note that there are several levels to what could be called the “recommended approach” in this paper. At the topmost level, it can be viewed merely as advocating the use of adaptive designs to assess the likelihood of future futility and efficacy of a clinical trial. But in illustrating that use, the authors seem also to advocate for adaptive methods resulting in sampling distributions that are less “heavy tailed” than analogous fixed sample designs (so that they can safely use naive analytic approaches), and they seem to fall prey to some of the difficulties in interpreting conditional power. We note that